Reducing Latency and Bandwidth Costs in Parallel Sparse Linear Solvers

Parallelizing sparse irregular application on distributed memory systems poses serious scalability challenges due to the communication bottlenecks which manifest themselves in an unpredictable manner as high bandwidth and/or latency overhead. The importance of different components of overall communication cost can be disproportionate due to the irregularity and sparseness inherent in the application. In such conditions, the best strategy for reducing communication overheads should favor the metric that is most crucial for the performance and a general method that attributes same importance to all metrics is likely to suffer. This work takes on the communication challenges offered by the latency-bound irregular applications, i.e., the applications characterized with high number of average and/or maximum messages per processor. The basic idea of our approach is to impose a regular communication pattern onto otherwise irregular communication operations and in this way to provide a low upper bound on the maximum number of messages handled by a processor. Using a regular communication pattern eliminates the irregularity in latency-bound communication operations and necessitates a store-and-forward scheme that consists of multiple stages of communication. Our findings show that the proposed approach is a remedy for the latency-bound applications; it scales seemingly unscalable instances and leads to an average of 50% reduction in parallel runtime on 256 processors.